Sim2Word: Explaining Similarity with Representative Attribute Words via Counterfactual Explanations

Recently, we have witnessed substantial success using the deep neural network in many tasks. Although there still exist concerns about explainability of decision making, it is beneficial for users to discern defects deployed models. Existing explainable models either provide image-level visualization attention weights or generate textual descriptions as post hoc justifications. Different from existing models, this article propose a new interpretation method that explains image similarity by salience maps and attribute words. Our model contains visual generation counterfactual explanation generation. The former has two branches: global identity relevant region discovery multi-attribute semantic discovery. first branch aims capture evidence supporting score, which achieved computing feature maps. second discover regions different attributes, helps understand attributes an might change score. Then, fusing branches, can obtain indicating important response evidence. latter will words best explain proposed erasing model. effectiveness our evaluated on classical face verification task. Experiments conducted benchmarks—VGGFace2 Celeb-A—demonstrate convincing interpretable explanations similarity. Moreover, algorithm be applied evidential learning cases, such finding most characteristic set images, verify its VGGFace2 dataset.